{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 2020百度之星·开发者大赛解决方案基线\n",
    "本项目基于深度学习平台飞桨（[PaddlePaddle](https://www.paddlepaddle.org.cn/)），利用一键式目标检测开发套件[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)与[度量学习库](https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/metric_learning)进行开发，欢迎使用并**star**。\n",
    "\n",
    "## 基线方案介绍\n",
    "\n",
    "本基线包含检测方案流程和匹配方案流程两部分。\n",
    "\n",
    "检测部分用于检测交通标志物的位置同时进行分类。基于飞桨推出的端到端目标检测开发套件[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)进行二次开发，PaddleDetection提供了丰富的检测模型，包含目标检测、实例分割、人脸检测等100+个预训练模型，涵盖多种数据集竞赛冠军方案，同时还具备了高灵活度，通过模块化的设计解藕各个组建，基于配置文件即可轻松搭建各种检测模型。本基线的检测部分支持了对交通标志物数据集读取和评估，采用faster_rcnn算法，ResNet50_vd作为主干网络，引入FPN模块，并对anchor的设置进行微调，在测试集上检测部分的mAP达到40.88。通过实验发现，本次任务检测的准确率对最终精度指标影响较大，因此可以选择PaddleDetection中提供的大量主干网络优化精度，也可以尝试PaddleDetection已具备的扩展特性进一步提升模型效果，例如group_norm, Modulated Deformable Convolution等，更多模型可以参考[MODEL_ZOO文档](https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.3/docs/MODEL_ZOO_cn.md)。\n",
    "\n",
    "匹配部分用于提取检测结果的图像特征，使得匹配的特征具有更大的相似度。基于飞桨提供的只能视觉工具PaddleCV中的[metric learning](https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/metric_learning)进行二次开发，本基线支持了交通标志物数据集读取和评估，首先对检测框进行细分类，采用resnet50算法和softmax loss进行训练；接着使用训练好的模型作为预训练模型，使用resnet50算法和triplet loss进行finetune，最终得到匹配结果在测试集上的F1 score达到0.466.\n",
    "\n",
    "\n",
    "\n",
    "| 基线方案 | 算法 | 精度 | 模型地址 | 备注 |\n",
    "| ------- | ---- | -------- | -------- | -------- |\n",
    "| 检测    | faster_rcnn + ResNet50_vd + FPN  | mAP=40.88   |  ```/home/aistudio/work/PaddleDetection_traffic/output/faster_rcnn_r50_vd_fpn_2x/best_model.pdparams```   |  基于[PaddleDetection](https://github.com/PaddlePaddle/PaddleDetection)进行二次开发  |\n",
    "| 匹配    |  细分类: resnet50 + softmax loss<br> finetune: resnet50 + triplet loss | F1 score=0.466 | 细分类: ```/home/aistudio/work/metric_learning_traffic/output_elem/ResNet50/180000.pdparams```<br> finetune: ```/home/aistudio/work/metric_learning_traffic/output_pair/ResNet50/10000.pdparams``` | 基于[metric learning](https://github.com/PaddlePaddle/models/tree/develop/PaddleCV/metric_learning)进行二次开发 |\n",
    "\n",
    "\n",
    "\n",
    "## 环境数据集和配置\n",
    "\n",
    "AIStudio上已安装PaddlePaddle 1.8.0，右下角可看到版本号和python环境"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio\n",
      "env: CUDA_VISIBLE_DEVICES=0\n"
     ]
    }
   ],
   "source": [
    "%cd ~/\n",
    "\n",
    "# 该数据集为Mini版本数据集\n",
    "#!mkdir data/traffic_data\n",
    "#!tar xvf data/data40016/mini_data.tar -C data/traffic_data\n",
    "\n",
    "# 该数据集为全量数据集\n",
    "# !unzip -q data/data42023/base_data.zip -d data\n",
    "# !mv data/base_data data/traffic_data\n",
    "%env CUDA_VISIBLE_DEVICES=0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "traffic数据组织方式如下：\n",
    "\n",
    "```txt\n",
    "data\n",
    "├── tag\n",
    "│   ├── test\n",
    "│   └── train\n",
    "├── train\n",
    "│   ├── pic\n",
    "│   └── input\n",
    "├── test\n",
    "│   ├── pic\n",
    "│   └── input\n",
    " ```\n",
    " \n",
    "  - **注意：** 数据集未划分验证集，获取数据集后需要选手自行划分\n",
    "\n",
    "## 检测方案流程\n",
    "### 1.配置路径"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/work/PaddleDetection_traffic\n"
     ]
    }
   ],
   "source": [
    "%cd /home/aistudio/work/PaddleDetection_traffic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 2.开始训练\n",
    "\n",
    "我们使用了COCO数据集上训练好的模型做迁移学习，基线提供的配置需要在V100单卡上训练12小时左右，更多训练参数请参考[检测库入门使用文档](https://github.com/PaddlePaddle/PaddleDetection/blob/release/0.3/docs/tutorials/GETTING_STARTED_cn.md)：\n",
    "\n",
    "**注意：**--eval为边训练边评估，基线中评估使用训练集，选手需要将validation数据集提前分好，存放在```/home/aistudio/data/traffic/val```和```/home/aistudio/data/traffic/tag/val```下,并修改配置文件中的image_dir和anno_path字段"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-19 00:48:25,249-INFO: If regularizer of a Parameter has been set by 'fluid.ParamAttr' or 'fluid.WeightNormParamAttr' already. The Regularization[L2Decay, regularization_coeff=0.000100] in Optimizer will not take effect, and it will only be applied to other Parameters!\n",
      "W0819 00:48:25.455018 11054 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0\n",
      "W0819 00:48:25.459581 11054 device_context.cc:260] device: 0, cuDNN Version: 7.3.\n",
      "2020-08-19 00:48:28,194-WARNING: /home/aistudio/.cache/paddle/weights/mask_rcnn_x101_vd_64x4d_fpn_1x.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]\n",
      "2020-08-19 00:48:29,112-WARNING: variable cls_score_b not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable rpn_cls_logits_fpn2_w not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable rpn_bbox_pred_fpn2_w not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable cls_score_w not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable rpn_cls_logits_fpn2_b not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable bbox_pred_b not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable bbox_pred_w not used\n",
      "2020-08-19 00:48:29,112-WARNING: variable rpn_bbox_pred_fpn2_b not used\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/io.py:1997: UserWarning: This list is not set, Because of Paramerter not found in program. There are: conv5_mask_w mask_inter_feat_4_w mask_inter_feat_3_w mask_inter_feat_1_w mask_inter_feat_2_w mask_inter_feat_2_b mask_inter_feat_4_b mask_fcn_logits_w conv5_mask_b mask_inter_feat_1_b mask_inter_feat_3_b mask_fcn_logits_b\n",
      "  format(\" \".join(unused_para_list)))\n",
      "2020-08-19 00:48:33,242-INFO: 80690 samples in file ../../data/traffic_data/tag/train/\n",
      "2020-08-19 00:48:33,248-INFO: places would be ommited when DataLoader is not iterable\n",
      "2020-08-19 00:48:35,536-INFO: iter: 0, lr: 0.001000, 'loss_cls': '2.944388', 'loss_bbox': '0.000003', 'loss_rpn_cls': '0.617506', 'loss_rpn_bbox': '0.002757', 'loss': '3.564654', time: 0.000, eta: 0:00:05\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!python -u tools/train.py \\\n",
    "    -c configs/traffic/resnext.yml \\\n",
    "    -o pretrain_weights=https://paddlemodels.bj.bcebos.com/object_detection/mask_rcnn_x101_vd_64x4d_fpn_1x.tar finetune_exclude_pretrained_params=[cls_score,bbox_pred] \\\n",
    "    #--eval"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 3.输出结果\n",
    "\n",
    "训练完成后，结果默认存放在```/home/aistudio/work/PaddleDetection_traffic/output/faster_rcnn_r50_vd_fpn_2x/```下，加载最终训练模型并对测试集数据进行'signs'字段的预测，结果默认保存在```/home/aistudio/work/PaddleDetection_traffic/output/detect```目录下，格式为单独json文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-08-19 21:45:28,047-INFO: 41125 samples in file ../../data/traffic_data/test/input/\n",
      "2020-08-19 21:45:28,049-INFO: places would be ommited when DataLoader is not iterable\n",
      "W0819 21:45:28.088201 12806 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0\n",
      "W0819 21:45:28.092785 12806 device_context.cc:260] device: 0, cuDNN Version: 7.3.\n",
      "2020-08-19 21:45:35,661-INFO: Test iter 0\n"
     ]
    }
   ],
   "source": [
    "!python tools/eval.py \\\n",
    "    -c configs/traffic/resnext.yml \\\n",
    "    -o weights=output/resnext/model_final.pdparams \\\n",
    "    save_prediction_only=true\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "同时也可计算检测算法在验证集上的mAP指标，**注意选手需要提前分好数据并把存放在```/home/aistudio/data/traffic/val```和```/home/aistudio/data/traffic/tag/val```下,并修改配置文件中的image_dir和anno_path字段，基线方案使用train作为评估阶段数据集**, 并使用如下方式进行评估：\n",
    "\n",
    "```python\n",
    "!python -u tools/eval.py \\\n",
    "    -c configs/traffic/faster_rcnn_r50_vd_fpn_2x.yml \\\n",
    "    -o weights=output/faster_rcnn_r50_vd_fpn_2x/best_model.pdparams\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 匹配方案流程\n",
    "\n",
    "该阶段对数据集中的’match‘字段进行学习，将group A和group B中检测到的标志物进行匹配，使得相同类别的标志物具有更高的特征相似度。\n",
    "\n",
    "### 1. 配置路径\n",
    "\n",
    "将目录切换至```metric_learning_traffic```中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/aistudio/work/metric_learning_traffic\n"
     ]
    }
   ],
   "source": [
    "%cd ~/work/metric_learning_traffic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 2. 开始训练\n",
    "\n",
    "匹配阶段分为两部：细分类训练和匹配训练。使用ResNet50作为骨干网络，预训练模型已经放置在```/home/aistudio/work/metric_learning_traffic/pretrain/ResNet50_pretrained/```中\n",
    "\n",
    "#### 细分类训练\n",
    "\n",
    "首先对数据集进行细分类训练，能够进行匹配的样本作为一类，在训练开始时也会对数据集匹配样本的类别进行统计，基线方案在V100单卡上预计训练9小时\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----------  Configuration Arguments -----------\n",
      "arc_easy_margin: False\n",
      "arc_margin: 0.15\n",
      "arc_scale: 80.0\n",
      "checkpoint: None\n",
      "class_dim: 9993\n",
      "data_path: ../../data/traffic_data/train\n",
      "display_iter_step: 20\n",
      "embedding_size: 0\n",
      "image_shape: 3,64,64\n",
      "loss_name: softmax\n",
      "lr: 0.01\n",
      "lr_steps: 9000,15000\n",
      "lr_strategy: piecewise_decay\n",
      "model: ResNet50\n",
      "model_save_dir: output_elem\n",
      "pretrained_model: pretrain/ResNet50_pretrained/\n",
      "save_iter_step: 1000\n",
      "total_iter_num: 18000\n",
      "train_batch_size: 256\n",
      "use_gpu: True\n",
      "------------------------------------------------\n",
      "W0819 20:48:59.598527 12200 device_context.cc:252] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.2, Runtime API Version: 9.0\n",
      "W0819 20:48:59.603104 12200 device_context.cc:260] device: 0, cuDNN Version: 7.3.\n",
      "2020-08-19 20:49:02,308-WARNING: pretrain/ResNet50_pretrained/.pdparams not found, try to load model file saved with [ save_params, save_persistables, save_vars ]\n",
      "warning: variable fc_0.b_0 not used\n",
      "warning: variable fc_0.w_0 not used\n",
      "total matched class number: 38468\n",
      "total instance number: 217774\n",
      "[2020-08-19 20:49:12] trainbatch 0, lr 0.010000, loss 9.386873, acc1 0.0000, acc5 0.0000, time 0.00 sec\n",
      "[2020-08-19 20:50:49] trainbatch 20, lr 0.010000, loss 9.410289, acc1 0.0000, acc5 0.0000, time 0.15 sec\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!python train_elem.py --pretrained_model=pretrain/ResNet50_pretrained/ --data_path=../../data/traffic_data/train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "更多细分类训练参数设置可以使用如下方式查看\n",
    "\n",
    "```python\n",
    "python train_elem.py -h\n",
    "```\n",
    "\n",
    "#### 匹配训练\n",
    "\n",
    "完成细分类训练后，基于训练好的模型进行finetune，通过triplet loss进行匹配训练, 基线方案在V100单卡上预计训练6小时"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 此处180000.pdparams可按需更改模型文件\r\n",
    "!python train_pair.py --pretrained_model=output_elem/ResNet50/180000.pdparams --data_path=../../data/traffic_data/train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "更多匹配训练参数设置可以使用如下方式查看\n",
    "\n",
    "```python\n",
    "python train_pair.py -h\n",
    "```\n",
    "\n",
    "### 3. 测试匹配结果\n",
    "\n",
    "在完成两阶段训练后，对匹配模型效果进行评估，评估使用验证集的图片和标注信息，根据标注好的'signs'信息预测得到'match'信息，最终结果默认保存在```/home/aistudio/work/metric_learning_traffic/output/result```中，可进一步通过评测脚本得到F1 score。\n",
    "\n",
    "测试阶段分为：对纯匹配模型的效果测试和结合检测结果的匹配效果进行测试\n",
    "\n",
    "#### 纯匹配模型的效果测试\n",
    "\n",
    "**注意选手需要提前分好数据并把存放在```/home/aistudio/data/traffic/val```和```/home/aistudio/data/traffic/tag/val```下**："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# validation为选手自行划分结果，包含标注结果，并将val路径替换train\n",
    "# 此处10000.pdparams可按需更改模型文件\n",
    "!python eval_pair.py --pretrained_model=output_pair/ResNet50/10000.pdparams --data_path=../../data/traffic_data/train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 结合检测结果的匹配效果进行测试\n",
    "\n",
    "该阶段将检测结果进行匹配并保存结果在```/home/aistudio/work/metric_learning_traffic/output/result```中，这种方式适用与最终test集的测试和结果提交，注意此时设置```detect_path```为检测阶段的输出结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 此处10000.pdparams可按需更改模型文件\r\n",
    "!python eval_pair.py --pretrained_model=output_pair/ResNet50/10000.pdparams --data_path=../../data/traffic_data/test --detect_path=../PaddleDetection_traffic/output/detect"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.8.0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
